…#43307)

While the #43260 attempted to address the problem where example
dag importability tests should skip provider tests on non-main,
it did not actually solve the problem.

While debugging it, it turned out that since #42505, the provider
tests were not executed in main "at all" - the "providers"
directory was not included in the list of places to check for
the example dags (they were in "airflow" in v2-10-test") this is
why it "looked like" the solution worked in "main".

This PR fixes both problems:

* brings back importability of provider's example_dags in main branch
* properly excludes the providers examples in non-main branch

Another teething problem after moving providers in apache#42505. After
moving providers, the history of the current folder in "providers"
only contains changes after the move - it does not include
changes from before the move - and since we always regenerate
the full list of commits - they were missing.

We cannot use `--follow` - because `git log --follow` only
works for single files, not directories, but since the move
was very predictable ("airflow/providers/nnn" ->
"airflow/providers/src/airflow/providers/nnn") we can add the old
path to `git log` command to get both - pre and post move commit
history.

Another teething problem after moving providers in apache#42505. After
moving providers, the history of the current folder in "providers"
only contains changes after the move - it does not include
changes from before the move - and since we always regenerate
the full list of commits - they were missing.

We cannot use `--follow` - because `git log --follow` only
works for single files, not directories, but since the move
was very predictable ("airflow/providers/nnn" ->
"airflow/providers/src/airflow/providers/nnn") we can add the old
path to `git log` command to get both - pre and post move commit
history.

Another teething problem after moving providers in #42505. After
moving providers, the history of the current folder in "providers"
only contains changes after the move - it does not include
changes from before the move - and since we always regenerate
the full list of commits - they were missing.

We cannot use `--follow` - because `git log --follow` only
works for single files, not directories, but since the move
was very predictable ("airflow/providers/nnn" ->
"airflow/providers/src/airflow/providers/nnn") we can add the old
path to `git log` command to get both - pre and post move commit
history.

After apache#42505, you need to get through extra hoops to develop
providers in Airflow's monorepo.

This is a simple (not yet uv-specific) documentation on how to
install providers in editable mode when you want to develop
providers, so that you can run unit test.

Copied mostly from apache#43082

It's not yet full set of docs explaining how to use workspaces
and UV. This shoudl be handled via  apache#43200

…3468)

* Add simple instructions for installing providers in editable mode

After #42505, you need to get through extra hoops to develop
providers in Airflow's monorepo.

This is a simple (not yet uv-specific) documentation on how to
install providers in editable mode when you want to develop
providers, so that you can run unit test.

Copied mostly from #43082

It's not yet full set of docs explaining how to use workspaces
and UV. This shoudl be handled via  #43200

* Apply suggestions from code review

Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>

---------

Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>

When providers have been moved in apache#42505 broke passing parameters
when provider tests were passed as extra args of "testing tests"
command of breeze.

Previously all tests were under "tests" folder and there was
an exclusion to disable "All" tests when any test was passed as
parameter. But after moving tests to "providers" this stopped working.

Additional exclusion needs to be added for "providers/tests" and
"providers/tests_sdk/".

This PR also adds autocompletion for tests passed this way by
setting the click type to Path for the extra args (but without
the need for the Path to exist).

Also during this check it turned out that "All" tests are not
working in the intended way - but this should not impact our CI
only local runs.

Appropriate comment has been added and it's captured in
apache#42632

When providers have been moved in apache#42505 broke passing parameters
when provider tests were passed as extra args of "testing tests"
command of breeze.

Previously all tests were under "tests" folder and there was
an exclusion to disable "All" tests when any test was passed as
parameter. But after moving tests to "providers" this stopped working.

Additional exclusion needs to be added for "providers/tests" and
"providers/tests_sdk/".

This PR also adds autocompletion for tests passed this way by
setting the click type to Path for the extra args (but without
the need for the Path to exist).

Also during this check it turned out that "All" tests are not
working in the intended way - but this should not impact our CI
only local runs.

Appropriate comment has been added and it's captured in
apache#42632

When providers have been moved in #42505 broke passing parameters
when provider tests were passed as extra args of "testing tests"
command of breeze.

Previously all tests were under "tests" folder and there was
an exclusion to disable "All" tests when any test was passed as
parameter. But after moving tests to "providers" this stopped working.

Additional exclusion needs to be added for "providers/tests" and
"providers/tests_sdk/".

This PR also adds autocompletion for tests passed this way by
setting the click type to Path for the extra args (but without
the need for the Path to exist).

Also during this check it turned out that "All" tests are not
working in the intended way - but this should not impact our CI
only local runs.

Appropriate comment has been added and it's captured in
#42632

The compatibility tests in CI are using providers built as packages
from sources, so the compatibility tests run there using
"providers/tests" work just fine, because all providers are installed
in the airflow.providers site library. However when we are iterating
and debugging backwards compatiblity provider tests, we should be
able to use local provider sources, rather than installed packages
and we have the possibility of mounting both - providers sources
and tests to the image.

See `contributing-docs/testing/unit_tests.rst` on how to do it
by using ``--mount-sources providers-and-tests`` flag connected
with `--use-airflow-version`.

However as of apache#42505 this has been broken, because currently in
main we  rely on airflow having "pkgutil" namespace package for
both - airflow, and airflow.providers packages (previous airflow
versions had implicit package for airflow.providers package) - so
providers installed locally cannot be used as "another" source of
providers.  Previously it was working because both "installed" and
"sources" `airflow.providers` package were implicit namespace packages.

As explained in https://packaging.python.org/en/latest/guides/packaging-namespace-packages/#native-namespace-packages

> Every distribution that uses the namespace package must include such
> an `__init__.py`. If any distribution does not, it will cause the
> namespace logic to fail and the other sub-packages will not be
> importable. Any additional code in __init__.py will be inaccessible.

So because old airflow uses implicit provider's packages and
main airflow from source uses "explicit" provider's package,
the only way we can make the "source" providers is to mount
them or symbolically link them to inside installed distribution
of airflow package (in site directory) (or dynamically remove
the __init__.py from provider's source directory.

We cannot mount the provider package sources ot inside the
installed airflow - because when --use-airflow-version is used,
airflow is installed dynamically inside the container - after
the container is started.

This PR solves the problem by adding an env variable that
will make the initialization script to remove the installed
airflow.providers folder after installing airflow and linking the
"providers/src/airflow/providers" folder there. This has the
added benefit that all providers (including the preinstalled ones)
are used from "main" sources rather than from installed packages -
which was problematic for the past way of using providers from
sources - which used the fact that both "airflow.providers" in
the site-library and the one in sources were implicit namespace
packages.

…ws (#43617)

* Enable back iterative development of latest providers with old airflows

The compatibility tests in CI are using providers built as packages
from sources, so the compatibility tests run there using
"providers/tests" work just fine, because all providers are installed
in the airflow.providers site library. However when we are iterating
and debugging backwards compatiblity provider tests, we should be
able to use local provider sources, rather than installed packages
and we have the possibility of mounting both - providers sources
and tests to the image.

See `contributing-docs/testing/unit_tests.rst` on how to do it
by using ``--mount-sources providers-and-tests`` flag connected
with `--use-airflow-version`.

However as of #42505 this has been broken, because currently in
main we  rely on airflow having "pkgutil" namespace package for
both - airflow, and airflow.providers packages (previous airflow
versions had implicit package for airflow.providers package) - so
providers installed locally cannot be used as "another" source of
providers.  Previously it was working because both "installed" and
"sources" `airflow.providers` package were implicit namespace packages.

As explained in https://packaging.python.org/en/latest/guides/packaging-namespace-packages/#native-namespace-packages

> Every distribution that uses the namespace package must include such
> an `__init__.py`. If any distribution does not, it will cause the
> namespace logic to fail and the other sub-packages will not be
> importable. Any additional code in __init__.py will be inaccessible.

So because old airflow uses implicit provider's packages and
main airflow from source uses "explicit" provider's package,
the only way we can make the "source" providers is to mount
them or symbolically link them to inside installed distribution
of airflow package (in site directory) (or dynamically remove
the __init__.py from provider's source directory.

We cannot mount the provider package sources ot inside the
installed airflow - because when --use-airflow-version is used,
airflow is installed dynamically inside the container - after
the container is started.

This PR solves the problem by adding an env variable that
will make the initialization script to remove the installed
airflow.providers folder after installing airflow and linking the
"providers/src/airflow/providers" folder there. This has the
added benefit that all providers (including the preinstalled ones)
are used from "main" sources rather than from installed packages -
which was problematic for the past way of using providers from
sources - which used the fact that both "airflow.providers" in
the site-library and the one in sources were implicit namespace
packages.

* Update Dockerfile.ci

Co-authored-by: GPK <gopidesupavan@gmail.com>

* Update scripts/docker/entrypoint_ci.sh

Co-authored-by: GPK <gopidesupavan@gmail.com>

---------

Co-authored-by: GPK <gopidesupavan@gmail.com>

This is a no-op change right now, but as part of the provider re-org in apache#42505
this sets us up to be able to load the providers code in the tests

The reason this change is done separately is that changes to breeze code form
forks doesn't take effect, and this small change makes it easier to land on
main without having to re-create that large PR.

…roject (apache#42505)

This is only a partial split so far. It moves all the code and tests, but
leaves the creation of `core/` to a separate PR as this is already large
enough.

In addition to the straight file rename the other changes I had to make here
are:

- Some mypy/typing fixes.

  Mypy can be fragile about what it picks up when, so maybe some of those
  changes were caused by that. But the typing changes aren't large.

- Improve typing in common.sql type stub

  Again, likely a mypy file oddity, but the types should be safe

- Removed the `check-providers-init-file-missing` check

  This isn't needed now that airflow/providers shouldn't exist at all in the
  main tree.

- Create a "dev.tests_common" package that contains helper files and common
  pytest fixtures

  Since the provider tests are no longer under tests/ they don't automatically
  share the fixtures from the parent `tests/conftest.py` so they needed
  extracted.

  Ditto for `tests.test_utils` -- they can't be easily imported in provider
  tests anymore, so they are moved to a more explicit shared location.

In future we should switch how the CI image is built to make better use of UV
caching than our own approach as that would remvoe a lot of custom code.

…roject (apache#42505) (apache#42624)

This is only a partial split so far. It moves all the code and tests, but
leaves the creation of `core/` to a separate PR as this is already large
enough.

In addition to the straight file rename the other changes I had to make here
are:

- Some mypy/typing fixes.

  Mypy can be fragile about what it picks up when, so maybe some of those
  changes were caused by that. But the typing changes aren't large.

- Improve typing in common.sql type stub

  Again, likely a mypy file oddity, but the types should be safe

- Removed the `check-providers-init-file-missing` check

  This isn't needed now that airflow/providers shouldn't exist at all in the
  main tree.

- Create a "dev.tests_common" package that contains helper files and common
  pytest fixtures

  Since the provider tests are no longer under tests/ they don't automatically
  share the fixtures from the parent `tests/conftest.py` so they needed
  extracted.

  Ditto for `tests.test_utils` -- they can't be easily imported in provider
  tests anymore, so they are moved to a more explicit shared location.

In future we should switch how the CI image is built to make better use of UV
caching than our own approach as that would remvoe a lot of custom code.

Co-authored-by: Ash Berlin-Taylor <ash@apache.org>
Co-authored-by: Ryan Hatter <25823361+RNHTTR@users.noreply.github.com>

* Move tests common without changes

* Fix docstrings in tests_common

* Move tests_common from "dev" to top-level.

Follow-up after apache#42505 fixing teething problem with tests_common.

Originally in apache#42505 common test code was moved to "dev" folder, but
the "dev" folder is really dedicated to "build" scripts and the
problem with moving "tests_common" to the folder was that the
whole "dev" folder is replaced (for non-committer PRs) with
the content from the target branch.

This is done for security reasons, because we can accidentally
use any of the scripts from dev in the CI build scripts and we
might not notice, which will open us to a security issue where
a file in "dev" coming from PR could be accidentally executed
during the "pull_request_target" workflow - which would expose
our secrets and GitHub Package write permissions to a
contributor coming from a fork.

This change moves the files, fixes pre-commit specification and
docs, also fixes a number of "doc" issues detected by "ruff" in
the tests_common folder as they were detected after the move.
The tests_common folder is added to folders mounted when
breeze is executed with local folders mounted (in order to
avoid accidental mounting of randomly generated files to
inside the breeze container).

All imports for the common tests were updated to reflect this
move.

…ache#43173)

Cleans-up airflow and providers `__init__.py" files in order to
get providers import work again.

This is done by excluding the two `__init__.py` files from
automated ruff isort rules adding `from __future__ import annotations`.

That should finally get rid of the Intellij teething import
problem that has been introduced in apache#42505.

There were earlier - unsuccessful - attempts to fix it in
the apache#43116 and apache#43081 that followed apache#42951 - but the key is that Pycharm
requires the namespace's extend_path to be first "real" line
of code in the `__init__.py` to understand that the package
is an "explicit" namespace package - and it conflicts with
the requirement of "from __future__ import annotations" to be
the first line of Python code.

Also this PR fixes following problem:

* pytest_plugin expecting .asf.yml in "airflow" sources - even during
  compatibility tests with older version of airflow (where the
  .asf.yml is not present)

---------

Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>

…43273)

While the apache#43260 attempted to address the problem where example
dag importability tests should skip provider tests on non-main,
it did not actually solve the problem.

While debugging it, it turned out that since apache#42505, the provider
tests were not executed in main "at all" - the "providers"
directory was not included in the list of places to check for
the example dags (they were in "airflow" in v2-10-test") this is
why it "looked like" the solution worked in "main".

This PR fixes both problems:

* brings back importability of provider's example_dags in main branch
* properly excludes the providers examples in non-main branch

Another teething problem after moving providers in apache#42505. After
moving providers, the history of the current folder in "providers"
only contains changes after the move - it does not include
changes from before the move - and since we always regenerate
the full list of commits - they were missing.

We cannot use `--follow` - because `git log --follow` only
works for single files, not directories, but since the move
was very predictable ("airflow/providers/nnn" ->
"airflow/providers/src/airflow/providers/nnn") we can add the old
path to `git log` command to get both - pre and post move commit
history.

…ache#43468)

* Add simple instructions for installing providers in editable mode

After apache#42505, you need to get through extra hoops to develop
providers in Airflow's monorepo.

This is a simple (not yet uv-specific) documentation on how to
install providers in editable mode when you want to develop
providers, so that you can run unit test.

Copied mostly from apache#43082

It's not yet full set of docs explaining how to use workspaces
and UV. This shoudl be handled via  apache#43200

* Apply suggestions from code review

Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>

---------

Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>

…43529)

When providers have been moved in apache#42505 broke passing parameters
when provider tests were passed as extra args of "testing tests"
command of breeze.

Previously all tests were under "tests" folder and there was
an exclusion to disable "All" tests when any test was passed as
parameter. But after moving tests to "providers" this stopped working.

Additional exclusion needs to be added for "providers/tests" and
"providers/tests_sdk/".

This PR also adds autocompletion for tests passed this way by
setting the click type to Path for the extra args (but without
the need for the Path to exist).

Also during this check it turned out that "All" tests are not
working in the intended way - but this should not impact our CI
only local runs.

Appropriate comment has been added and it's captured in
apache#42632

…ws (apache#43617)

* Enable back iterative development of latest providers with old airflows

The compatibility tests in CI are using providers built as packages
from sources, so the compatibility tests run there using
"providers/tests" work just fine, because all providers are installed
in the airflow.providers site library. However when we are iterating
and debugging backwards compatiblity provider tests, we should be
able to use local provider sources, rather than installed packages
and we have the possibility of mounting both - providers sources
and tests to the image.

See `contributing-docs/testing/unit_tests.rst` on how to do it
by using ``--mount-sources providers-and-tests`` flag connected
with `--use-airflow-version`.

However as of apache#42505 this has been broken, because currently in
main we  rely on airflow having "pkgutil" namespace package for
both - airflow, and airflow.providers packages (previous airflow
versions had implicit package for airflow.providers package) - so
providers installed locally cannot be used as "another" source of
providers.  Previously it was working because both "installed" and
"sources" `airflow.providers` package were implicit namespace packages.

As explained in https://packaging.python.org/en/latest/guides/packaging-namespace-packages/#native-namespace-packages

> Every distribution that uses the namespace package must include such
> an `__init__.py`. If any distribution does not, it will cause the
> namespace logic to fail and the other sub-packages will not be
> importable. Any additional code in __init__.py will be inaccessible.

So because old airflow uses implicit provider's packages and
main airflow from source uses "explicit" provider's package,
the only way we can make the "source" providers is to mount
them or symbolically link them to inside installed distribution
of airflow package (in site directory) (or dynamically remove
the __init__.py from provider's source directory.

We cannot mount the provider package sources ot inside the
installed airflow - because when --use-airflow-version is used,
airflow is installed dynamically inside the container - after
the container is started.

This PR solves the problem by adding an env variable that
will make the initialization script to remove the installed
airflow.providers folder after installing airflow and linking the
"providers/src/airflow/providers" folder there. This has the
added benefit that all providers (including the preinstalled ones)
are used from "main" sources rather than from installed packages -
which was problematic for the past way of using providers from
sources - which used the fact that both "airflow.providers" in
the site-library and the one in sources were implicit namespace
packages.

* Update Dockerfile.ci

Co-authored-by: GPK <gopidesupavan@gmail.com>

* Update scripts/docker/entrypoint_ci.sh

Co-authored-by: GPK <gopidesupavan@gmail.com>

---------

Co-authored-by: GPK <gopidesupavan@gmail.com>